There have been many cases where I know I started lower than some of the same people claiming that, worked harder, and achieved more, and yet I'm still not ignorant enough to deny that all that is still just a coinflip and with one minor changed circumstance, I could have ended up on the streets or something. I think a big part of it is that, starting so "low" on the totem pole, you tend to meet more peers who have as much or more potential than you growing up, who never realized it just due to the wrong thing at the wrong time.
That's very true. Because our best opportunity lay in working hard and working smart does not mean working hard and working smart guarantees a good opportunity. Life is a cold, hard thing. Individuals are born into this world selfish and for the most part we go through life with a strong sense of self-preservation (and gratification!) as a first priority.
Collectively humans have, over the years, attempted to construct a wide variety of social spheres called "government" aimed at improving opportunities for whatever citizens are party to that sphere. But it remains the case that no social sphere (read: government in this case) is able to deliver "fair" to everyone's satisfaction for an infinite number of reasons, one being that what is perceived as fair to one person/community is not perceived as fair by another person/community.
Our personal best option lay in opting to live under whatever government we can that, in our view, represents the most in terms of "fair" and then choose to live there, participate as a citizen there and work with whatever that system offers as best we can.